Know some strategies for clarifying conceptual definitions and writing conceptual definitions
Defining terms
Research Questions
Does religion play a role in civil wars?
What causes increasing income inequality in least developed countries?
Why has public opinion on same sex marriage liberalized so rapidly?
Is the U.S. becoming more polarized?
Not everyone is going to agree on what these terms mean!
Before we can get anywhere researching these questions, we need to define the thing we’re studying
Defining things: harder than it seems!
Reasonable definition of a chair
Chair
Terms
Conceptual Definition
A description of the concrete, measurable properties of a concept and the unit of analysis to which it applies
Unit of Analysis
The entity that that is being studied. For instance: individuals, governments, parties etc.
Operational Definition
A description of the instrument used to measure the concept
Crafting Conceptual Definitions: Unit of Analysis
What is the unit? What entity possesses the characteristic?
The U.S. and Canada are democracies (concept: democracy, unit: countries)
Lincoln, LBJ, and Trump are the three tallest U.S. presidents (concept: height, unit: presidents/people)
Labour is the UK’s main center-left political party (concept: party ideology/family, unit: political party)
One way to think of this: if you put your data into a spreadsheet, what does each “row” represent? The answer to that question is the unit of analysis.
Unit of analysis and the ecological fallacy
The ecological fallacy occurs when you attribute a characteristic of groups to individuals. For instance:
High-income states tend to vote for Democrats, but rich people tend to vote for Republicans.
Suicide rates in 19th century Europe were higher in Protestant countries compared to Catholic countries, but Protestant individuals were no more likely to die by suicide.
In the 1930 census, zip codes with more immigrants had higher literacy rates, but immigrant individuals had lower literacy rates.
All of these are fundamentally problems of ignoring the unit of analysis.
Crafting Conceptual Definitions: Key features
What are the essential features of the concept?
If there are cases or definitions everyone agrees on, what characteristics do they share?
Are certain qualities necessary or sufficient for a case to belong to a category?
What characteristics are most distinctive to those cases?
What is most helpful for clarifying edge-cases?
Are there multiple dimensions or just one? If two characteristics always occur together, you might only need to account for one of them!
Crafting Conceptual Definitions: transparency
We want to avoid being Potter Stewart:
The goal is cumulative knowledge, and a private definition can’t provide that.
Some steps to take:
Identify the unit of analysis
Make a list of important properties, clear examples and non-examples, and/or generally accepted definitions
Remove items that aren’t measurable
Reduce dimensions where possible (if a characteristic is shared by positive cases and negative cases, then it isn’t useful)
Refine as needed
Conceptualizing Democracy
Some proposed features:
Regular elections with meaningful alternatives
Peaceful transfer of power
Free expression
A competitive media environment
Autonomous political groups
Rule of law
Checks and balances
Property rights
Conceptualizing Democracy
Unit of analysis: governments (usually national governments)
Definitely autocracies: North Korea, Saudi Arabia, Russia
Edge cases: Hungary, Turkey, Tunisia
Features:
Voting and elections
Multiple parties
Meaningful civil liberties
Stability and monopoly on violence
This may need refinement: what about sham elections?
This may not be distinctive: there are stable autocracies on the list
Operational Definitions
Operationalization
Even where we agree on a definition, we will need to measure a concept and there is often slippage here
Considerations: Parsimony
In many cases, we trade some truth for simplicity. All else equal, a simpler definition is better than a more complicated one.
Considerations: Parsimony
This is a very accurate rendering of the DC metro area
But this is probably more useful for getting around.
Considerations: Parsimony
Similarly: we might consider “democracy” to be a highly complex multidimensional concept
mindmap
root((Democracy))
Liberal Rights
Free Speech
Free Association
Rule of law
Deliberation
Respectful Dialogue
Adversarial Press
Broad Participation
Egalitarianism
Equal Rights
Equitable access to resources
Diverse representation
Majoritarianism
Fair elections
Peaceful transfer of power
Meaningful alternatives
Considerations: Parsimony
…but we might still prefer an operational definition that relies on a subset of concepts that are easy to measure.
mindmap
root((Democracy))
Majoritarianism
Fair elections
Peaceful transfer of power
Meaningful alternatives
Democracy along multiple dimensions. Scores are determined by:
Polling a large group of area experts on a range of democratic dimensions (electoral, deliberative, liberal etc.)
Using an algorithm to aggregate those responses into scores along each dimension.
Considerations: Reliability
Reliability refers to how consistently the same measurement instrument produces the same result
Measuring reliability
Test-Retest method: give the same test to the same group twice. Measure the correlation between answers.
Measuring reliability
Zaller, J., & Feldman, S. (1992). A simple theory of the survey response: Answering questions versus revealing preferences. American journal of political science, 579-616.
Measuring reliability
Split half: for a measurement using a multi-item scale, we can split results and use half the questions to predict the remaining half.
If the scale questions all measure the same “thing” then you would expect the halves to resemble each other.
Considerations: Validity
Validity refers to whether an operational definition correctly measures the concept we’re interested in.
A challenge to validity might come from cases that seem mis-categorized in our definition. For example: the minimalist definition of democracy suggests that Japan was authoritarian for much of the latter half of the 20th century.
Considerations: Validity and Reliability
Measures could be reliable but invalid: your astrological sign is a reliable measure of your political views because it doesn’t change, but it also isn’t valid because it has no real correspondence to your actual politics.
Measures could be valid but unreliable: an exam with a single well-written question might be a valid measure of a student’s understanding of material, but it will probably have more random variability compared to an exam with multiple poorly-written questions.
Measurement Error
Random Error
Radio interference
Data entry errors
Random non-response
Systematic Error
Hawthorne effect (people behave differently when they know they’re being observed)
Social desirability bias (people don’t want to admit to doing bad stuff, even on an anonymous survey
Systematic non-response (surveys) or missing data (everything)
Measurement Error
Measurement error is a source of noise, systematic measurement error is a source of bias
We’ll generally find that random noise is much easier to deal with than bias because we can simply collect more data. Significant bias, on the other hand, presents profound challenges to research because we can only correct it if we can measure it
Measuring Party ID
Consider measuring party identification:
Instrument
Problem
Registration
Excludes new/non-voters, people in states without partisan registration, and people who register one way and vote another
Policy views
Complicated to measure, and a surprising number of people don’t have consistent policy views!
Voting Behavior
Excludes new/non-voters, may not be consistent even in a single election, people may exaggerate or forget voting behavior.
Self Description*
Subjective, far more self-described “independents” than people who consistently vote that way
Party ID
Sources like the American National Election Study rely on self-description to measure party ID, but each respondent receives two questions instead of one.
Party ID
Roughly a third of the electorate describe themselves as “independent”
Party ID
When pressed, many independents will say they lean toward a party.
Party ID
Based on voting behavior, “leaners” are more similar to partisans.
So we commonly measure party ID with two questions.
Party ID
The ANES method for measuring party ID is hardly the only option we have, but it has some nice features:
It’s relatively reliable and survey respondents understand it
It’s close to the conceptual definition of party ID as a sort of self-identity or group affinity.
It’s transparent and widely used
It limits the influence of social desirability bias by giving respondents lots of options
It’s linked to an observed behavior: vote choice
Key points
Asking social science questions requires us to define complex ideas and measure them. And we can lose a lot in that process.
There are some wrong answers, but there is generally no single correct answer. We generally have to make trade-offs between things we might value like:
Parsimony
Reliability
Validity
Probability of random or systematic errors
We can’t really achieve perfection, but where there is disagreement, we want to be transparent about methods and limitations.